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Abstract. We present a Projection onto Convex Sets (POCS) type algorithm for 
solving systems of linear equations. POCS methods have found many applications 
ranging from computer tomography to digital signal and image processing. The 
Kaczmarz method is one of the most popular solvers for overdetcrmincd systems 
of linear equations due to its speed and simplicity. Here we introduce and analyze 
an extension of the Kaczmarz method that iteratively projects the estimate onto 
a solution space given by two randomly selected rows. We show that this projec- 
tion algorithm provides exponential convergence to the solution in expectation. 
The convergence rate improves upon that of the standard randomized Kaczmarz 
method when the system has correlated rows. Experimental results confirm that in 
this case our method significantly outperforms the randomized Kaczmarz method. 



1. Introduction 

We consider a consistent system of linear equations of the form 

Ax = b, 

where b e C' m and A G C mxn is a full-rank m x n matrix that is overdetermined, 
having more rows than columns (m > n). When the number of rows of A is large, 
it is far too costly to invert the matrix to solve for x, so one may utilize an iterative 
solver such as the Projection onto Convex Sets (POCS) method, used in many 
applications of signal and image processing [H [18]. The Kaczmarz method is often 
preferred, iteratively cycling through the rows of A and orthogonally projecting the 
estimate onto the solution space given by each row [TU]. Precisely, let us denote 
by a 1 , a 2 , . . ., a m the rows of A and b\, b 2 , ■ ■ ., b m the coordinates of b. We 
assume each pair of rows is linear independent, and for simplicity, we will assume 
throughout that the matrix A is standardized, meaning that each of its rows has unit 
Euclidean norm; generalizations from this case will be straightforward. Given some 
trivial initial estimate Xo, the Kaczmarz method cycles through the rows of A and 
in the kth iteration projects the previous estimate x k onto the solution hyperplane 
of (di, x) = bi where i = k mod m, 

x k+1 = x k + (bi - (di, x k ))a,i. 
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Theoretical results about the rate of convergence of the Kaczmarz method have 
been difficult to obtain, and most are based on quantities which are themselves hard 
to compute [3j [TJ. Even more importantly, the method as we have just described 
depends heavily on the ordering of the rows of A. A malicious or unlucky order- 
ing may therefore lead to extremely slow convergence. To overcome this, one can 
select the rows of A in a random fashion rather than cyclically [HI EE2]- Strohmer 
and Vershynin analyzed a randomized version of the Kaczmarz method that in each 
iteration selects a row of A with probability proportional to the square of its Eu- 
clidean norm [201 US]- Thus in the standardized case we consider, a row of A is 
chosen uniformly at random. This randomized Kaczmarz method is described by 
the following pseudocode. 

Algorithm 1.1: Randomized Kaczmarz 
Input: Standardized matrix A, vector b 

Output: An estimation Xk of the unique solution x to Ax = b 

Set x . { Trivial initial approximation } 

k <- 

repeat 

k <- k + 1 

Select r 6 {1, 2, . . . , n} { Randomly select a row of A } 

Set £Efc «— a3fc_! + {b r — (a r , Xk_ 1 ))a r { Perform projection } 

Note that this method as stated selects each row with replacement, see [17] for 
a discussion on the differences in performance when selecting with and without 
replacement. Strohmer and Vershynin show that this method exhibits exponential 
convergence in expectation [20| [T9j. 



Here and throughout, || • H2 denotes the vector Euclidean norm, || ■ || denotes the 
matrix spectral norm, || • \\p denotes the matrix Frobenius norm, and the inverse 
||A _1 || = inf{M : M||Aa;|| 2 > \\x\\2 for all x} is well-defined since A is full-rank. 
This bound shows that when A is well conditioned, the randomized Kaczmarz 
method will converge exponentially to the solution in just O(n) iterations (see 
Section 2.1 of [20] for details). The cost of each iteration is the cost of a single 
projection and takes 0(n) time, so the total runtime is just 0(n 2 ). This is superior 
to Gaussian elimination which takes 0(mn 2 ) time, especially for very large systems. 
The randomized Kaczmarz method even substantially outperforms the well-known 
conjugate gradient method in many cases [20] . 

Leventhal and Lewis show that for certain probability distributions, the expected 
rate of convergence can be bounded in terms of other natural linear-algebraic quanti- 
ties. They propose generalizations to other convex systems [IT] . Recently, Chen and 

2 



(1.1) 




Powell proved that for certain classes of random matrices A, the randomized Kacz- 
marz method convergences exponentially to the solution not only in expectation but 
also almost surely |16j . 

In the presence of noise, one considers the possibly inconsistent system Ax + w ~ b 
for some error vector w. In this case the randomized Kaczmarz method converges 
exponentially fast to the solution within an error threshold |13j . 



where R the the scaled condition number as in ( II .ip and || • ||oo denotes the largest 
entry in magnitude of its argument. This error is sharp in general [13]. Modified 
Kaczmarz algorithms can also be used to solve the least squares version of this 
problem, see for example [H El El |2] and the references therein. 

1.1. Coherent systems. Although the convergence results for the randomized 
Kaczmarz method hold for any consistent system, the factor in the convergence 
rate may be quite small for matrices with many correlated rows. Consider for exam- 
ple the reconstruction of a bandlimited function from nonuniformly spaced samples, 
as often arises in geophysics as it can be physically challenging to take uniform sam- 
ples. Expressed as a system of linear equations, the sampling points form the rows 
of a matrix A; for points that are close together, the corresponding rows will be 
highly correlated. 

To be precise, we examine the pairwise coherence of a standardized matrix A by 
defining the quantities 

(1.2) A = A(A) = max \(aj, a,k)\ and 8 — 5(A) — min \{a,j, Ofc)|. 



Remark. These quantities measure how correlated the rows of the matrix A are. 
We point out that this notion of coherence coincides with that of signal processing 
terminology and is different than the alternative definition which measures the cor- 
relation between singular vectors and the canonical vectors. The notion of coherence 
used here simply gives a measure of pairwise row correlation. Analysis using the 
notion of coherence for singular vectors may also lead to improved convergence rates 
for these methods, and we leave this as future work. 

Note also that because A is standardized, < 8 < A < 1. It is clear that when A 
has high coherence parameters, ||A _1 || is very small and thus the factor R in ( II. ip 
is also small, leading to a weak bound on the convergence. Indeed, when the matrix 
has highly correlated rows, the angles between successive orthogonal projections are 
small and convergence is stunted. We can explore a wider range of orthogonal direc- 
tions by looking towards solution hyperplanes spanned by pairs of rows of A. We 
thus propose a modification to the randomized Kaczmarz method where each iter- 
ation performs an orthogonal projection onto a two-dimensional subspace spanned 
by a randomly-selected pair of rows. We point out that the idea of projecting in 




fc/2 
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each iteration onto a subspace obtained from multiple rows rather than a single row 
has been previously investigated numerically, see e.g. [6j[l|. 



With this as our goal, a single iteration of the modified algorithm will consist of the 
following steps. Let x k denote the current estimation in the kth iteration. 

• Select two distinct rows a r and a s of the matrix A at random 

• Compute the translation parameter e 

• Perform an intermediate projection: y <— x k + s(b r — (xk, a r ))a r 

• Perform the final projection to update the estimation: Xk+i <— y + (b s — 
(y,a s ))a s 

In general, the optimal choice of e at each iteration of the two-step procedure cor- 
responds to subtracting from Xy. its orthogonal projection onto the solution space 
{x : (a r ,x) = b r and (a s ,x) = b s }, which motivates the name two-subspace Kacz- 
marz method. By optimal choice of e, we mean the value e opt minimizing the residual 
1 1 a; — ajfe+illjj. Expanded, this reads 

\\x - x k +i\\l = ||e(6 r - (x k ,a r ))(a r - (a g ,a r ) a s ) + x k - x + (b s - {x k ,a s ))a s \\ 2 2 . 

Using that the minimizer of ||7to + z\\\ is 7 = — , we see that 

- (a r - (a 8 , Or) a s , x k - x + (b s - (x k , a s ))a s ) 
° pt (b r ~ (x k ,a r ))\\a r - (a s ,a r )a s \\l 

Note that the unknown vector x appears in this expression only through its observ- 
able inner products, and so e opt is computable. After some algebra, one finds that 
the two-step procedure with this choice of e op t can be re-written in the following 
numerically stable formulation. 

Algorithm 1.2: Two-subspace Kaczmarz 



Input: Matrix A, vector b 

Output: An estimation x k of the unique solution x to Ax = b 



Set Xq. { Trivial initial approximation } 

k <- 

repeat 

k <- k + 1 

Select r,se{l,2,...,n}{ Select two distinct rows of A uniformly at random } 
Set /ifc ^— (a r , a s ) { Compute correlation } 

Set y k 4— x k _x + (b s — (xk—i, a s ))a s { Perform intermediate projection } 
Set v k <— ar ~Vka,s r Q om p U te vector orthogonal to a s in direction of a r } 

Set f3 k < br ^ bs ^ { Compute corresponding measurement } 

>/ 1 — l/j-fc I 2 



x k <- y k + {/3 k - (yk, v k ))v k " { Perform projection } 



We note that by the assumption that each pair of rows is linearly independent, we 
have \fik\ 7^ 1 for all k so that division is always well-defined. Our main result 
shows that the two-subspace Kaczmarz algorithm provides the same exponential 
convergence rate as the standard method in general, and substantially improved 
convergence when the rows of A are coherent. Figure [T] plots two iterations of the 
one-subspace random Kaczmarz algorithm and compares this to a single iteration 
of the two-subspace Kaczmarz algorithm. 




Figure 1 . For coherent systems, the one-subspace randomized Kacz- 
marz algorithm (a) converges more slowly than the two-subspace 
Kaczmarz algorithm (b). 

Theorem 1.1. Let A be a full-rank standardized matrix with n columns andm > n 
rows and suppose Ax = b. Let x^ denote the estimation to the solution x in the 
kth iteration of the two-subspace Kaczmarz method. Then 

- x k \\l < {(l - ||a? — acolll, 

where D = min | S , A 1 ^ A ' ) j , A and 5 are the coherence parameters (II .2|) , and 
R = || A |||, II A -1 1| 2 denotes the scaled condition number. 

Remarks. 1. When A = 1 or 5 = we recover the same convergence rate as 
provided for the standard Kaczmarz method (11. ip since the two-subspace method 
utilizes two projections per iteration. 

2. The bound presented in Theorem ll.ll is a pessimistic bound. Even when A = 1 or 
6 = 0, the two-subspace method improves on the standard method if any rows of A 
are highly correlated (but not equal). This is evident from the proof of Theorem ll.il 
in Section [3] via Lemma 13.11 but we present the bound for simplicity. Under other 
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assumptions on the matrix A, improvements can be made to the convergence bound 
of Theorem ll.il For example, if one assumes that the correlations between the rows 
are non-negative, one obtains the bound 

1,2 (f iV D E \ k II „2 
E \\X — Xh 9 < 1 — \\x — Xn L, 

11 kU2 - \ v ^/ r qj 

where E = 4<5 3 and Q = \\Q~~ 1 \\ 2 \\Q\\'p is the scaled condition number of the m 2 x n 
matrix ft whose rows consist of normalized row differences from A, aj — a,i. See [T5] 
for details and the proof of this result. 

3. Theorem 11.11 yields a simple bound on the expected runtime of the two-subspace 
randomized Kaczmarz method. To achieve accuracy e, meaning 

E||£Cfc — x\\\ < e 2 \\x — a? || 25 

one asks that 

21oge 



E(fc) < 



log ((1 " " S) ' 
If A is well-conditioned then R = 0(n) and we thus require that 

2n 



O 



Since each iteration requires 0(n) time, for large enough n this again yields a total 
runtime of 0(n 2 ) as in the standard randomized Kaczmarz case [20], but with an 
improvement in the constant factors. 

4. When the rows of A have arbitrary norms, one may simply select pairs of rows 
uniformly at random, normalize prior to performing the projections, and obtain the 
result of Theorem 11.11 in terms of the standardized matrix. One obtains an alterna- 
tive bound by selecting pairs of distinct rows a r and a s with probability proportional 
to the product || Qf> 1 1 2 1 1 ^ s 

|| 2, following the strategy of Strohmer and Vershynin [20] 
in the standard randomized Kaczmarz algorithm. Defining the normalized variables 
a r = a r /\\a r \\ 2 and b r = b r /\\a r \\2, the algorithm proceeds as before with these 
substitutions in place. We define the coherence parameters fll.2p in terms of the 
normalized rows, and we define a new matrix A := DA, where D is a diagonal 
matrix with entries Djj = — ||aj ll^) 1 ^ 2 - Then one follows the proof of Theo- 

rem [LTJ to obtain the analogous convergence bound in the non- standardized case, 



E\\x - x k \\l < [ [ 1 - Vj -S] ||a; — asolll 

where D is as in Theorem [LTJ and R = \\A\\ 2 F \\A~ l \\ 2 denotes the scaled condition 
number of A. 
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Figure [2] shows the value of D of Theorem 11.11 for various values of A and 5. This 
demonstrates that the improvement factor D is maximized when 6 = A = ^ ~ 
0.62, giving a value of D m 0.1. 




Figure 2. A plot of the improved convergence factor D as a function 
of the coherence parameters 5 and A > 5. 



1.2. Organization. Next in Section [2] we present some numerical results demon- 
strating the improvements offered by the two-subspace randomized Kaczmarz method 
We then prove our main result, Theorem 11.11 in Section [3j We end with a brief dis- 
cussion in Section HI 

2. Numerical Results 

In this section we perform several experiments to compare the convergence rate 
of the two-subspace randomized Kaczmarz with that of the standard randomized 
Kaczmarz method. As discussed, both methods exhibit exponential convergence in 
expectation, but when the rows of the matrix A are coherent, the two-subspace 
method exhibits much faster convergence. 

To test these methods, we construct various types of 300 x 100 matrices A. To 
acquire a range of 5 and A, we set the entries of A to be independent identically 
distributed uniform random variables on some interval [c, 1] . Changing the value 
of c will appropriately change the values of 5 and A. Note that there is nothing 
special about this interval, other intervals (both negative and positive or both) 
of varying widths yield the same results. For each matrix construction, both the 
randomized Kaczmarz and two-subspace randomized methods are run with the same 
fixed initial (randomly selected) estimate and fixed matrix. The estimation errors for 
each method are computed at each iteration and averaged over 40 trials. The heavy 
lines depict the average error over these trials, and the shaded region describes the 
minimum and maximum errors. Since each iteration of the two-subspace method 
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utilizes two rows of the matrix A, we will equate a single iteration of the standard 
method with two iterations in Algorithm 11.11 for fair comparison. 

Figure [3] demonstrates the regime where the two-subspace method offers the most 
improvement over the standard method. Here the matrix A has highly coherent 
rows, with 8 ~ A. 




Figure 3. A log-linear plot of the error per iteration for the random- 
ized Kaczmarz (RK) and two-subspace RK (2SRK) methods. Matrix 
A has highly coherent rows, with entries uniformly distributed on 
[0.9, 1] yielding 5 = 0.998 and A = 0.999. 

Our result Theorem 1 1 . 1 1 suggest s that as 5 becomes smaller the two-subspace method 
should offer less and less improvements over the standard method. When 5 = the 
convergence rate bound of Theorem 1 1.1 1 is precisely the same as that of the standard 
method (11.11) . Indeed, we see this precise behavior as is depicted in Figure HI 

3. Main Results 

We now present the proof of Theorem 11.11 We first derive a bound for the expected 
progress made in a single iteration. Since the two row indices are chosen indepen- 
dently at each iteration, we will be able to apply the bound recursively to obtain 
the desired overall expected convergence rate. 

Our first lemma shows that the expected estimation error in a single iteration of the 
two-subspace Kaczmarz method is decreased by a factor strictly less than that of 
the standard randomized method. 

Lemma 3.1. Let x k denote the estimation to the solution of Ax = b in the kth iter- 
ation of the two-subspace Kaczmarz method. Denote the rows of A by a 1 ,a 2 , . . . a m . 
Then we have the following bound, 

E||cc-cc fe ||2 < ( 1 - — ) Hsc-Xfc-illa 5- y^ C^ s ((x-x k _ 1 ,a r ) 2 + (x-x k _ 1 ,a s ) 2 ), 

v ' r<s 
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Figure 4. A log-linear plot of the error per iteration for the random- 
ized Kaczmarz (RK) and two-subspace RK (2SRK) methods. Matrix 
A has entries uniformly distributed on [c, 1] with coherence param- 
eters (a) 5 = 0.937 and A = 0.986 (c = 0.5), (b) 5 = 0.760 and 
A = 0.954 (c = 0.2), (c) 5 = 0.394 and A = 0.870 (c = -0.1), and (d) 
5 = and A = 0.740 (c = -0.5). 



where C rs = W : a{ f" 
condition number. 



//.., 



(a r , a s ), and R 



\A 1 || 2 ||A|||. denotes the scaled 



Proof. We fix an iteration k and for convenience refer to v^, fi^, and as v, /i, and 
y, respectively. We will also denote 7 = (a r , v). 

First, observe that by the definitions of v and Xy. we have 

x k = a? fc _i + (x - x k _!, a s )a s + (x - x k _ x , v)v. 

Since a s and v are orthonormal, this gives the estimate 



(3.1) \\x - x k \\l = \\x - x k _ x \\l - \{x - x k _^a s )\ 2 - \{x - x k _^v)\ 2 



We wish to compare this error with the error from the standard randomized Kacz- 
marz method. Since we utilize two rows per iteration in the two-subspace Kaczmarz 
method, we compare its error with the error from two iterations of the standard 



method. Let z and z' be two subsequent estimates in the standard method follow- 
ing the estimate Xk-i, and assume z ^ z' . That is, 

(3.2) z = £C fe _i + (b r - (x k -i,a r ))a r and z' = z + (b s - (z, a 8 ))a 8 . 

Recalling the definitions of v, fi and 7, we have 

(3.3) a r = fia s + 71? with ji 2 + r ) 2 = 1. 

Substituting this into ( 13. 2 p yields 

z = x k _! + fi(x - x k _!, a r )a s + j{x - x k _^ a r )v. 

Now substituting this into (I3.2p and taking the orthogonality of a s and v into 
account, 

z' = x k _ x + {x- cc fe _i, a s )a s + ^{x - x k _ 1: a r )v. 

For convenience, let e k _x = x — x k _x denote the error in the (k — l)st iteration of 
two-subspace Kaczmarz. Then we have 

\\ x ~ z '\\l = \\ e k-i - (e k -i, a s )a s - 7(e fc _i, a r )v\\l 

= \\e k -i - (e fc _i, a s )a s - (e fc _i, v)v - (7(e fc _i, a r ) - (e k -i, v))v\\l 
= \\e k -i\\l - |(e fc _i,a s )| 2 - |(e fc _i,i;)| 2 + \^y{e k -i,a r ) - (e fc _i,i;)| 2 . 

The third equality follows from the ortho normality of a s and v. We now expand 
the last term, 

|7<e fc _i, o,.) - (e fc _i, v) | 2 = |7(e fc _i, fia B + -yv) - (e fe _i, v) | 2 

= |7 2 (e fe _i,v) +j/i(e k - 1 ,a a ) - (e fc _i,i;)| 2 
= |yU 2 (e fc _i,t>) - 7/i(e fe _i,a s )| 2 . 

This gives 

Hcc-z'H 2 = II II I - |(e fc _ 1 ,a s )| 2 - \(e k -x, v)\ 2 + \n 2 {e k _i,v) - 7/x(e fc _i, a s ) | 2 . 

Combining this identity with (13.11) . we now relate the expected error in the two- 
subspace Kaczmarz algorithm, E||aj — x k \\ 2 to the expected error of the standard 
method, E|| follows: 

(3.4) E||a; - ccfc|| 2 = E||a; - z'\\l - E\fi 2 (e k _ 1 ,v) - 7/i(e fc _i, a s )| 2 . 

It thus remains to analyze the last term. Since we select the two rows r and s 
independently from the uniform distribution over pairs of distinct rows, the expected 
error is just the average of the error over all m 2 — m ordered choices r, s. To this 

10 



end we introduce the notation /jL r)S = (a r , a s ). Then by definitions of v, \x and 7, 
E|/i 2 (e fc _i, v) - 7/x(e fc _i, a s )| 2 



1 


m 2 




m 




1 




m 2 




m 




1 





((e fc _i, a r ) - /i r , s (e fe _i, a s )) - /i r , n /l - // 2 (e fe _i, a s ) 



1 — /z 



— m 



\ e fc— I; Otr) 



/V,s 



1 /i 2 , 



Hr >8 Jl-Hl (e fc _i,a fl ) 



2 

r.s 



(e fc _i, a s ) 



We now recall that for any 6,ir,u, and i>, 

(flu - H 2 + (0u - ttm) 2 > (|tt| - |0|) V + ^ 2 ). 

2 

Setting r)S = , r,3 2 and 717^ = / Mr,s 2 , we have by rearranging terms in the 
symmetric sum, 

E|/i 2 (e fc _i, 0) - 7/x(e fc _i, a s )| 2 



(3.5) 



1 


m 2 




m 




1 




m 2 




m 




1 




m 2 




m 




1 




m 2 




rn 



r<s 



r<s 



r<s 



2 

/ \f^r,8 1 AV s \ 2 / / \ 9 / \ 9\ 

7= b ' ) ((e fc _i,a r ) +(e fc _i,a fl )j. 

v v 1 - w,- y 



Since selecting two rows without replacement (i.e. guaranteeing not to select the 
same row back to back) can only speed the convergence, we have from fll.il) that 
the error from the standard randomized Kaczmarz method satisfies 

E||se - z'\\l < (1 - l/R) 2 \\x - x k -i\\l. 
Combining this with H3.4[) and ( 13. 5 p yields the desired result. 

□ 
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Although the result of Lemma 13.11 is tighter, the coherence parameters 5 and A 
of f ll.2j) allow us to present the following result which is not as strong but simpler 
to state. 

Lemma 3.2. Let x k denote the estimation to Ax = b in the kth iteration of the 
two-subspace Kaczmarz method. Denote the rows of A by a±, a 2 , . . . a m . Then 

E\\X - X k \\l < ^1 - -^J - ||CC - |||, 

where D = min | S , A 1 ^ A ^ \, 5 and A are the coherence parameters as in (11.21) . 
and R = || A _1 || 2 || A|||i denotes the scaled condition number. 

Proof. By Lemma [3. II we have 

E||a5— aSfcUl < (l - — J ||a3— £C fc _i||| — V] C 2 S Ux - x k -i, a r ) 2 + (x - x k _ x , a s ) 2 ) 

\ R I m A — m z — ' ' 

v 7 r<s 

where 

_ \{a r ,a„)\ - (a r ,a s ) 2 



1 - (a r , a s ) 2 

By the assumption that 5 < \(a r , a s ) \ < A, we have 

C 2 Mnini ^ 1 -^ p 
C r , s >mmi , 



Thus we have that 

EC 2 S Ux - x k -!, a r ) 2 + {x- x k -i, a s ) 2 ) 



m 2 — m 



r<s 



> — «^ — y^((x- x k _x, a r ) 2 + (x- x k _!, a s ) 2 ) 
m — m — 4 

r<s 



D(m-r 



m 2 — m 



^2(x - Xk^Ctr)' 



r=l 

II* - : 

(3-6) > - 



D ||as-aj fc _i||| 



m \\A~ l \\1 

In the last inequality we have employed the fact that for any 2, 

m \\z\\ 2 

r=l " " 2 

Combining (13. 6 p and ( 13. 61) along with the definition of R yields the claim. 

□ 
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Applying Lemma I3~2l recursively and using the fact that the selection of rows in each 
iteration is independent yields our main result Theorem 11.11 



4. Conclusion 

As is evident from Theorems ll.l| the two-subspace Kaczmarz method provides ex- 
ponential convergence in expectation to the solution of Ax = b. The constant in 
the rate of convergence for the two-subspace Kaczmarz method is at most equal to 
that of the best known results for the randomized Kaczmarz method (II. ip . When 
the matrix A has many correlated rows, the constant is significantly lower than that 
of the standard method, yielding substantially faster convergence. This has positive 
implications for many applications such as nonuniform sampling in Fourier analysis, 
as discussed in Section [TJ 

We emphasize that the bounds presented in our main theorems are weaker than 
what we actually prove, and that even when 5 is small, if the rows of A have many 
correlations, Lemma 13.11 still guarantees improved convergence. For example, if the 
matrix A has correlated rows but contains a pair of identical rows and a pair of or- 
thogonal rows, it will of course be that 5 = and A = 1. However, we see from the 
lemmas in the proofs of our main theorems that the two-subspace method still guar- 
antees substantial improvement over the standard method. Numerical experiments 
in cases like this produce results identical to those in Section [21 

It is clear both from the numerical experiments and Theorem 11.11 that the two- 
subspace Kaczmarz performs best when the correlations (a r , a s ) are bounded away 
from zero. In particular, the two-subspace method offers the most improvement 
over the standard method when 5 is large. The dependence on A, however, is 
not as straightforward. Theorem 11.11 suggests that when A is very close to 1 the 
two-subspace method should provide similar convergence to the standard method. 
However, in the experiments of Section [2] we see that even when Awl, the two- 
subspace method still outperforms the standard method. This exact dependence on 
A appears to be only an artifact of the proof. 

4.1. Extensions to noisy systems and higher subspaces. As is the case for 
many iterative algorithms, the presence of noise introduces complications both the- 
oretically and empirically. We show in [TB] that with noise the two-subspace method 
provides expected exponential convergence to a noise threshold proportional to the 
largest entry of the noise vector w. A further and important complication that 
noise introduces is semi-convergence, a well-known effect in Algebraic Reconstruc- 
tion Technique (ART) methods (see e.g. [5]). It remains an open problem to deter- 
mine the optimal stopping condition without knowledge of the solution x. See [15] 
for more details. Alternatively, the optimal trade-off between speed and accuracy 
may be reached by employing a hybrid Kaczmarz algorithm which initially imple- 
ments two-subspace Kaczmarz iterations to reach an approximate solution quickly, 
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but switches to standard Kaczmarz iterations after a certain number of iterations 
to arrive at a more accurate final approximation. 

Finally, a natural extension to our method would be to use more than two rows 
in each iteration. Indeed, extensions of the two-subspace algorithm to arbitrary 
subspaces can be analyzed [13] . 



References 

C. Cenker, H.G. Feichtinger, M. Mayer, H. Steier, and T. Strohmer. New variants of the POCS 
method using afhne subspaces of hnitc codimension, with applications to irregular sampling. 
In Conf. SPIE 92 Boston, pages 299-310, 1992. 

Y. Censor, P.P.B. Eggermont, and D. Gordon. Strong underrelaxation in Kaczmarz's method 
for inconsistent systems. Numer. Math., 41(l):83-92, 1983. 

F. Deutsch and H. Hundal. The rate of convergence for the method of alternating projections. 
J. Math. Anal. AppL, 205(2):381-405, 1997. 

P. Drineas, M.W. Mahoney, S. Muthukrishnan, and T. Sarlos. Faster least squares approxi- 
mation. Numerische Mathematik, 117(2):217-249. 

T. Elfving, T. Nikazad, and P. C. Hansen. Semi-convergence and relaxation parameters for a 
class of SIRT algorithms. Electron. T. Numer. Ana., 37:321-336, 2010. 

H. G. Feichtinger and T. Strohmer. A Kaczmarz-based approach to nonperiodic sampling on 
unions of rectangular lattices. In Proc. Conf. SampTA-95, pages 32-37, 1995. 
A. Galantai. On the rate of convergence of the alternating projection method in finite dimen- 
sional spaces. J. Math. Anal. AppL, 310(l):30-44, 2005. 

M. Hanke and W. Niethammer. On the acceleration of Kaczmarz's method for inconsistent 
linear systems. Linear Alg. AppL, 130:83-98, 1990. 

G. T. Herman and L.B. Meyer. Algebraic reconstruction techniques can be made computation- 
ally efficient. IEEE T. Med. Imaging, 12(3):600-609, 1993. 

S. Kaczmarz. Angenaherte auflosung von systemen linearer gleichungcn. Bull. Internat. Acad. 
Polon.Sci. Lettres A, pages 335-357, 1937. 

D. Levcnthal and A.S. Lewis. Randomized methods for linear constraints: Convergence rates 
and conditioning. Math. Oper. Res., 35(3):641-654, 2010. 

F. Natterer. The Mathematics of Computerized Tomography. Wiley, New York, 1986. 

D. Needell. Randomized Kaczmarz solver for noisy linear systems. BIT Num. Math., 

50(2):395-403, 2010. 

D. Needell and J. A. Tropp. Paved with good intentions: Analysis of a randomized block 
kaczmarz method. Submitted, 2012. 

D. Needell and R. Ward. Two-subspace projection method for coherent overdetermined sys- 
tems. Technical report, Claremont McKenna College, 2012. 

A. Powell and X. Chen. Almost sure convergence for the Kaczmarz algorithm with random 
measurements. Submitted, 2012. 

B. Recht and C. Re. Beneath the valley of the noncommutativc arithmetic-geometric mean 
inequality: Conjectures, case studies, and consequences. Submitted for publication, 2012. 

K. M. Sezan and H. Stark. Applications of convex projection theory to image recovery in 
tomography and related areas. In H. Stark, editor, Image Recovery: Theory and application, 
pages 415-462. Acad. Press, 1987. 

T. Strohmer and R. Vershynin. A randomized solver for linear systems with exponential con- 
vergence. In RANDOM 2006 (10th International Workshop on Randomization and Computa- 
tion), number 4110 in Lecture Notes in Computer Science, pages 499-507. Springer, 2006. 
T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential conver- 
gence. J. Fourier Anal. AppL, 15:262-278, 2009. 



14 



